HAProxy

The Reliable, High Performance TCP/HTTP Load Balancer

News

Sep, 18th, 2024 : HAProxyConf 2025 Call for Papers

The next edition of the HAProxyConf will be held on June 4-5 2025, with some workshops on June 3. The

Call for Papers is open

visit the conference's site

May, 29th, 2024 : HAProxy 3.0.0 release

HAProxyTech's blog

mailing-list announcement

Dec, 5th, 2023 : HAProxy 2.9.0 release

HAProxyTech's blog

mailing-list announcement

May, 31th, 2023 : HAProxy 2.8.0 release

HAProxyTech's blog

more synthetic mailing-list announcement

February, 14th, 2023 : CVE-2023-25725 fixed!

here on the mailing list announce

December, 1st, 2022 : HAProxy 2.7.0 release

announnce

HAProxyTech blog article

June, 16th, 2022 : HAProxyConf: Call for Papers

HAProxyConf 2022

simplified form

May, 31st, 2022 : HAProxy 2.6.0 release

detailed here

Mar, 26th, 2022 : QUIC experimentation

https://interop.seemann.io/

From a technical point, the way it's done is by having a separate haproxy process listening to QUIC on UDP port 1443, and forwarding HTTP requests to the existing process. The main process constantly checks the QUIC one, and when it's seen as operational, it appends an Alt-Svc header that indicates the client that an HTTP/3 implementation is available on port 1443, and that this announce is valid for a short time (we'll leave it to one minute only so that issues can resolve quickly, but for now it's only 10s so that quick tests cause no harm):

    http-response add-header alt-svc 'h3=":1443"; ma=60' if { var(txn.host) -m end haproxy.org } { nbsrv(quic) gt 0 }

As such, compatible browsers are free to try to connect there or not. Other tools (such as git clone) will not use it. For those impatient to test it, the QUIC process' status is reported at the bottom of the stats page here: http://stats.haproxy.org/. The "quic" socket in the frontend at the top reports the total traffic received from the QUIC process, so if you're seeing it increase while you reload the page it's likely that you're using QUIC to read it. In Firefox I'm having this little plugin loaded: https://addons.mozilla.org/en-US/firefox/addon/http2-indicator/. It displays a small flash on the URL bar with different colors depending on the protocol used to load the page (H1/SPDY/H2/H3). When that works it's green (H3), otherwise it's blue (H2). For Chrome there is HTTP Indicator which does the same but displays an orange symbol when using H3. Chrome only accepts H3 on port 443 (which we enabled as well for it). Note that H2 and H3 are only served when the site is browsed in HTTPS at https://haproxy.org/.

At this point I'd still say "do not reproduce these experiments at home". Amaury and Fred are still watching the process' traces very closely to spot bugs and stop it as soon as a problem is detected. But it's still too early for being operated by non-developers. The hope is that by 2.6 we'll reach the point where enthousiasts can deploy a few instances on not-too-sensitive sites with sufficient confidence and a little dose of monitoring.

Nov, 23th, 2021 : HAProxy 2.5.0 release

detailed here

November 5th, 2021 : Cool hardware donation from Zerodha

Just after the HAProxy 2.4.0 release in May this year, Kailash Nadh, CTO of Zerodha contacted me and offered to donate a pair of SolidRun HoneyComb LX2 boards to help us continue to improve out threading scalability. With 16 ARM cores, 32GB RAM, 64GB eMMC, 4x10GbE, and PCI-e in a tiny and quiet form factor, there definitely is plenty of potential to emphasize the cost of contention and to further improve our data model and our algorithms! As usual with SolidRun, the hardware design is really awesome (it feels like you're installing a PC, and they're about the only ones featuring the console port). However I must say I was really disappointed by their complete absence of communication during the 5 months it took to ship the board. Vishnu of Zerodha and me probably asked them for status updates 8 times in total and each time we were told "we'll contact you when it ships, no need to ask all the time". Last time we were told the parcel had been sent two weeks ago to DHL who had no way to contact me... no comment! Emailing a tracking number doesn't cost much, especially when their own tracking site still shows the order as "processing" after I received it! Hopefully SolidRun manages to address this deficiency in the future because right now they're among the very few who make affordable server-grade development boards and I hope to continue to be able to order their great products without trouble... Anyway, more news later when the tests really begin! For now the boards boot and are fast (hint: haproxy-2.5 builds in 6 sec). Many big thanks to Kailash and Vishnu of Zerodha for this very kind contribution!

November 3rd, 2021 : HAProxyConf 2021 is in less than two weeks!

https://www.haproxyconf.com/

speakers

presentations

May, 14th, 2021 : HAProxy 2.4.0 release

announce here

April 8th, 2021 : Broke new performance record: 2 million req/s & 100 Gbps

2 million HTTPS requests forwarded per second, and 100 Gbps of bandwidth

July 7th, 2020 : HAProxy 2.2.0 is ready!

announce here

November 25th, 2019 : HAProxy 2.1.0 is out!

announce here

September 9th, 2019 : HAProxyConf

HAProxyConf

June 21th, 2019 : Call for papers slightly extended

HAProxyConf 2019

June 16th, 2019 : HAProxy 2.0.0 is out!

2.0.0

read this announcement

send your proposals for the HAProxyConf 2019

May 22th, 2019 : HAProxyConf 2019 - Call for Papers

which will take place in Amsterdam on November 12 and 13.
We are designing this conference with our community in mind. All sessions will be highly informative and technical in nature, without marketing fluff. There will be a heavy focus on presentations from end-users, who will share their use cases and real-world recommendations. We also expect the conference to be a fun time and a great opportunity to connect with several hundred of your peers from across Europe and beyond.
We very much want our community members to play a big role in the conference. With that in mind, please consider submitting an abstract as part of our call-for-papers process, which is now open. You can submit your abstracts by visiting the conference website. Please note that any submissions that are obviously vendor or product pitches will not be considered by the content selection committee. Abstracts much be submitted by June 21, 2019. All chosen presenters will receive a free, full conference pass.
If you have any questions about the conference or the call-for-papers process, please contact us at submission@haproxy.com.

December 19th, 2018 : HAProxy 1.9.0 released

the announcement here

the detailed blog article

November 26th, 2017 : HAProxy 1.8.0 released

the announcement here

June 23th, 2017

version table

February 28th, 2017 : 1.7.3

the announcement here

here

December 25th, 2016 : 1.6.11 and 1.5.19

full 1.6.11 announcement here

1.5.19 announcement here

Code and changelogs are available here for 1.6 and here for 1.5 as usual.

December 13th, 2016 : 1.7.1

read the announcement here

Code and changelog are available here as usual.

November 25th, 2016 : HAProxy 1.7.0 now released!

consult the announcement here

upgraded to version 1.7.0

StartSSL

Code and changelog are available here.

November 20th, 2016 : 1.6.10

available here

Code and changelog are available here as usual.

November 10th, 2016 : 1.7-dev6

Stream Processing Offload Engine

Scientiamobile

available here

Code and changelog are available here as usual.

August, 14th, 2016 : 1.6.8 and 1.7-dev4

Among the issues fixed, we can cite the occasional segfault hitting a few users of zlib (it was the result of a misunderstanding of the API apparently), a segfault when using sc_trackers with a different table, a potential memory corruption when using "sni" on the server line, a few crashes in Lua's txn_done() called by actions and fetchers, a risk to freeze some random file descriptors after an attempt to pass data to a file descriptor being experiencing a connection retry explaining some observed zombie connections, some peers protocol encoding issues explaining some abnormal values and synchro losses, a problem with incorrect sample duplication during processing which could lead certain fetchers to fail to work as a stick table key, and I think that's about all for this time. Code and changelog are available here as usual.

For 1.7-dev4, we had quite some new stuff. One change that may affect some users is that we removed the magic consisting in assigning a server's check port to the same port as the first port of the first "bind" directive in the listener if any. It doesn't make sense at all, is not documented, doesn't work in many situations (eg: unix sockets) and makes it impossible to improve the configuration. Normally nobody uses this anymore since 1.6 due to the fact that it is not allowed anymore to specify a port on the "listen" line.

Developers may notice that now everything is rebuilt when they modify a ".h" file. Just do like me, append "DEP=" to your make command line and it will continue to work as before. Non-developer users are protected against easy mistakes and we are not bothered by a dependency hell.

A number of build fixes for OpenBSD were merged. In fact it would not build anymore since 1.6 due to various missing includes (or include order). It's now OK. I'm surprized that we didn't receive any complaint in one year, in the past people would report OpenBSD breakage. Maybe these users are now on FreeBSD which seems to work very well.

There were other updates like "set-src-port", "set-dst", "set-dst-port" actions, to force the incoming src/dst address/port to be replaced by the one in argument (useful for logging and also to force a connection to go to a server configured as 0.0.0.0). Another new action is the "track-sc" for http-response. This is nice to for counting certain response events. The "show tls-keys" CLI command can now display the current secrets. There were some filter changes. The SNI filters now support multicerts (rsa/ecdsa). We can also decode the Netscaler's CIP protocol which is an alternative to haproxy's PROXY protocol. We now have a few new sample fetch functions reporting various TCP-level information on Linux, FreeBSD and NetBSD such as RTT, number of retransmits, etc. It can make logs more usable during troubleshooting. And finally the command-line "-f" argument now supports directories in addition to file names. Files are loaded in alphabetical order. It is convenient for certain users, but beware of the orderning, use at your own risk!

Code and changelog are available here as usual.

April, 13th, 2016 : 1.5.17

here

March, 14th, 2016 : 1.7-dev2, 1.6.4, 1.5.16, 1.4.27, 1.3.28 (EOL)

A number of important bugs were fixed since last releases. Some of them impact 1.6 when using http-reuse (orphaned connections). A few Lua bugs were fixed as well, one of them causing a segfault and another one dead connections. Sample fetch functions were protected against misuse of layer 7 in tcp connection rules causing a segfault. And session variables could also be improperly used in connection rules with the same effect. For the less important fixes, some race conditions were addressed in the systemd wrapper possibly causing the orphaned processes some people were experiencing. In 1.5 there were two issues with idle keep-alive timeout handling on the server side, sometimes causing some short-time busy polling loops and sometimes causing a new incoming request to be aborted on a persistent connection (that browsers had to resend).

In 1.7-dev we've got the filters which introduce a number of hooks to plug some code in a more flexible way than analysers. The compression code was already adapted to use them. More to come later, possibly traffic shaping. The stats have been improved : now everything available in HTML is also available in the CSV output, and there is a new "typed" output format that is more friendly to aggregators. It is now possible to manipulate environment variables from within the config files, this will solve the problem people are facing when migrating to systemd since it doesn't allow reloaded processes to see changes in environment variables.

As it's been a long time for all versions, users are encouraged to upgrade. Code and changelogs are available here as usual.

November, 1st, 2015 : 1.5.15

here

October, 20th, 2015 : 1.6.1

here

October, 13th, 2015 : HAProxy 1.6.0 now released!

The next release date for 1.7.0 is set to end of September 2016, or approximately one year. This time, in order to satisfy more contributors, we'll have a 3-phase development cycle. The first phase ending in March 2016 will merge the most sensitive changes, possibly causing a lot of breakage. It is only for developers. A second phase, ending in June, will be dedicated to fixing the breakage and will still allow small improvements to be made as long as they are not expected to cause regressions. It is possibly where we will decide to revert some of the early breakage if some features are too broken. Enthousiasts may start to test during this phase and report issues. The last phase ending in September will be dedicated to the final polishing, portability issues and doc updates, and should be acceptable for most early adopters. So let's get back to the whiteboards now.

September, 28th, 2015 : 1.6-dev6

the mailing list's announce

here

August, 30th, 2015 : 1.6-dev4

the mailing list's announce

here

July, 3rd, 2015 : 1.5.14 : fixes an information leak vulnerability (CVE-2015-3281)

here

June, 26th, 2015 : 1.5.13

here

June, 17th, 2015 : 1.6-dev2

May, 15th, 2015 : HTTP/2 is out!

RFC7540

RFC7541

May, 2nd, 2015 : 1.5.12

"option http-ignore-probes"

here

April, 1st, 2015 : April Fool's: Complete rewrite of HAProxy in Lua

the code base is increasing and is becoming slower to build day after day. Ten years ago, version 1.1.31 was only 6716 lines everything included. Today, mainline is 108395 lines, or 16 times larger.
gcc is getting slower over time. Since version 2.7.2 I used to rely on ten years ago, we've seen important slowdowns with v2.95, several v3.x then v4.x. I'm currently on 4.7 and afraid to upgrade.
while the whole code base used to build in less than a second ten years ago on an Athlon XP-1800, now it takes about 10 seconds on a core i5 at 3 GHz. Multiply this by about 200 builds a day and you see that half an hour is wasted every single day dedicated to development. That's about 1/4 of the available time if you count the small amount of time available after processing e-mails.
people don't learn C anymore at school and this makes it harder to get new contributors. In fact, most of those who are proficient in C already have a job and little spare time to dedicate to an opensource project.

In parallel, I'm seeing I'm getting old, I turned 40 last year and it's obvious that I'm not as much capable of optimizing code as I used to be. I'm of the old school, still counting the CPU cycles it takes a function to execute, the nanoseconds required to append an X-Forwarded-For header or to parse a cookie. And all of this is totally wasted when people run the software in virtual machines which only allocate portions of CPUs (ie they switch between multiple VMs at high rate), or install it in front of applications which saturate at 100 requests a second.

Recently with the Lua addition, we found it to be quite fast. Maybe not as fast as C, but Lua is improving and C skills are diminishing, so I guess that in a few years the code written in Lua will be much faster than the code we'll be able to write in C. Thus I found it wise to declare a complete rewrite of HAProxy in Lua. It comes with many benefits.

First, Lua is easy to learn, we'll get many more developers and contributors. One of the reason is that you don't need to care about resource allocation anymore. What's the benefit of doing an strdup() to keep a copy of a string when you can simply do "a = b" without having to care about the memory used behind. Machines are huge nowadays, much larger than the old Athlon XP I was using 10 years ago.

Second, Lua doesn't require a compiler, so we'll save 30 minutes a day per 200 builds, this will definitely speed up development for each developer. And we won't depend on a given C compiler, won't be subject to its bugs, and more importantly we'll be able to get rid of the few lines of assembly that we currently have in some performance-critical parts.

Third, last version of HAProxy saw a lot of new sample fetch functions and converters. This will not be needed anymore, because the code and the configuration will be mixed together, just as everyone does with Shell scripts. This means that any config will just look like an include directive for the haproxy code, followed by some code to declare the configuration. It will then be possible to create infinite combinations of new functions, and the configuration will have access to anything internal to HAProxy.

In the end, of the current HAProxy will only remain the Lua engine, and probably by then we'll find even better ones so that haproxy will be distributed as a Lua library to use anywhere, maybe even on IoT devices if that makes sense (anyone ever dreamed of having haproxy in their watches ?).

This step forward will save us from having to continue to do any code versionning, because everyone will have his own fork and the code will grow much faster this way. That also means that Git will become useless for us. In terms of security, it will be much better as it will not be possible to exploit a vulnerability common to all versions anymore since each version will be different.

HAProxy Technologies is going to assign a lot of resources to this task. Obviously all the development team will work on this full time, but we also realize that since customers will not be interested in the C version anymore after this public announce, we'll train the sales people to write Lua as well in order to speed up development.

We'll continue to provide an enterprise version forked from HAPEE that we'll rename "Luapee". It will still provide all the extras that make it a professional solution such as VRRP, SNMP etc and over the long term we expect to rewrite all of these components in Lua as well.

The ALOHA appliances will change a little bit, they'll mostly be a Lua engine to run all that code, so we'll probably rename them HALUA. And given that the appliance's goal has always been to take profit of the hardware and kernel to further improve the capabilities, we'll have free hands to port other performance-critical parts in Lua, including maybe the currently aging Linux kernel which also happens to be written in C.

Once everything is ported, I intend to use my old skills in the domain of microarchitecture to design a native Lua processor that will run in our appliances so that all the code runs in silicon and ends up being much faster than what we currently have in C.

I'm quite aware that some parts will be tedious. Rewriting OpenSSL in Lua will neither be easy nor fun. But it's the price to pay to get fast and affordable security.

Due to the huge amount of work, we'll postpone the 1.6 release to 1st April 2016, which leaves us exactly 366 days to complete this task. I hope everyone understands that we have no other choice.

February, 1st, 2015 : 1.5.11, 1.4.26, 1.3.27, 1.3.15.14

"http-request set-header"

"log-tag"

"http-send-name-header"

"show sess"

the 1.3.15 branch is closed and switches to the unmaintained state

Note:

git.haproxy.org

I had to perform a forced push

January, 1st, 2015 : Year of a changing Web

IETF HTTP working group

current form

"soon"

"around a few weeks"

What will this change ? Probably not much at the beginning, but a lot soon. It will take some time before web site operators figure the performance benefits brought by HTTP/2, but the media will quickly boast its merits and the change can happen quickly, even if just to catch up with competiting early adopters. A number of sites already support SPDY for the same reasons right now but SPDY is constantly evolving and requires more attention from users who have to update often. By being a new standard, HTTP/2 will not require that level of care, and it will be perceived as the direct descendant of HTTP/1.1, which is why it will be more adopted than SPDY.

All major browsers already support HTTP/2, two of them (Firefox and Chrome) will only support it for HTTPS. Internet Explorer will drop SPDY support once HTTP/2 is adopted. That logically means that a number of web sites will decide to enable HTTPS in order to support HTTP/2 for the users of the two aforementionned browsers. HTTPS comes with an extra round trip at the beginning of the connection, but HTTP/2 saves a lot of them during the transfers so in the end if there are at least a few tens of objects to retrieve, it will still be an improvement.

But this will cause a new issue : migrating to HTTPS will mean that the caches that are operated in universities, enterprises, all mobile phone operators and many ISPs will not be used anymore. This will immediately have two impacts : the first one is that the traffic on the internet will increase. Alarmists used to say that the 40 Tbps trans-atlantic total capacity is almost saturated and hard to upgrade, we'll see if that's true. The second effect is that origin servers will get a significant traffic increase, which is good for ADC vendors as well as for CDNs which will get many new customers and increase their revenue. Sadly, in a number of poorly connected countries where client-side caches are critical to the survival of the Internet, CDNs will not be able to help and the situation will get even worse. That's also the case for a number of mobile phone operators who can observe high cache hit ratios today.

What will very likely happen to address these situations is that ISPs and mobile phone operators will start to propose a faster Internet access to their customers in exchange for a root cert that they can happily install in their browser so that the operator can decipher SSL traffic on the fly and cache again. End users are already prepared to accept this because they don't care at all about their privacy when it comes to whatever they do with their smartphone, otherwise they would always close their apps and type a password to access their emails. And the next logical step is that mobile phones sold by these operators will already have the root cert pre-installed in order to save a complex operation from the end user.

And that will lead to an interesting situation. First, SSL offloading solution vendors will happily see their sales increase. But the counter part is that the chain of trust of the SSL/TLS model will be definitely broken in that there will be no way for the end user to know if his data were safe or not. This chain is extremely fragile already and is regularly being abused, but now it could become the norm not to trust SSL anymore when rogue CAs becomes mandatory to access the net.

Fortunately, a few solutions are being worked on. On the HTTP working group they're called "Trusted Proxies" or "GET https://", as a reference to what an HTTPS request through an explicit proxy could look like. They consist in letting the end user choose what can be deciphered and what cannot. That allows proxy operators to let some trusted sites pass through and to decipher/inspect/cache contents for all other ones. That's how we could get a better Internet for everyone, with better caching and better privacy at the same time. Not sure it will happen by 2015 though, but we should do whatever we can for this to happen!

December, 31th, 2014 : 1.5.10 : Last release of the year!

"option tcp-check"

here

November, 25th, 2014 : 1.5.9

here

October, 31th, 2014 : 1.5.8

here

October, 30th, 2014 : 1.5.7

here

October, 18th, 2014 : 1.5.6

here

October, 8th, 2014 : 1.5.5

here

September, 2nd, 2014 : 1.5.4

critical bug

CVE-2014-6269

tcp-request content track-sc

here

July 25th, 2014 : 1.5.3

here

July 12th, 2014 : 1.5.2

here

June 24th, 2014 : 1.5.1

Version 1.5.1 fixes a few bugs from 1.5.0 among which a really annoying one which can cause some file descriptor leak when dealing with clients which disappear from the net, resulting in the impossibility to accept new connections after some time. This bug was introduced in 1.5-dev25, so affected users are strongly encouraged to upgrade. For more information, please consult the source code and changelog here.

Also today I was pleased to receive a bottle of Champagne sent by our friends at Loadbalancer.org! Thank you guys!

June 19th, 2014 : HAProxy 1.5.0 released!

After 4 years of hard work, HAProxy 1.5.0 is finally released!

For people who don't follow the development versions, 1.5 expands 1.4 with many new features and performance improvements, including native SSL support on both sides with SNI/NPN/ALPN and OCSP stapling, IPv6 and UNIX sockets are supported everywhere, full HTTP keep-alive for better support of NTLM and improved efficiency in static farms, HTTP/1.1 compression (deflate, gzip) to save bandwidth, PROXY protocol versions 1 and 2 on both sides, data sampling on everything in request or response, including payload, ACLs can use any matching method with any input sample maps and dynamic ACLs updatable from the CLI stick-tables support counters to track activity on any input sample custom format for logs, unique-id, header rewriting, and redirects, improved health checks (SSL, scripted TCP, check agent, ...), much more scalable configuration supports hundreds of thousands of backends and certificates without sweating.

Since dev26, a few bugs were fixed, and some low-importance things were integrated. Basic OCSP stapling support from Dirkjan and Emeric was finally merged. Sasha's header replace actions were merged as well. I've added a few more info in the stats page (avg response times) and CSV output (health check status), added support for PROXY v2 on the accept side, and added the "capture" action on tcp-request in order to log contents such as SNI or payload. R�mi's dh-param was finally integrated.

People love numbers, so here are a few. From 1.4.0 to 1.5.0, we had :

1574 calendar days (4 yr 3 mon)
26 development versions (one every 2 months on average)
540 bugs fixed (387 added during 1.5, 153 affecting 1.4 as well)
2549 commits
683 unique commit dates (at least this many days worked)
up to 24 commits per day
69712 lines removed, 122279 lines added
many extremely useful bug reports (too many to list)
73 code/doc contributors : Adrian Bridgett, Alex Davies, Aman Gupta, Andreas Kohn, Apollon Oikonomopoulos, Arnaud Cornet, Baptiste Assmann, Bertrand Jacquin, Bhaskar Maddala, Conrad Hoffmann, Cyril Bont�, Daniel Schultze, David BERARD, David Cournapeau, Dave McCowan, David du Colombier, Delta Yeh, Dirkjan Bussink, Dmitry Sivachenko, Emeric Brun, Emmanuel Hocdet, Evan Broder, Finn Arne Gangstad, Gabor Lekeny, Geoff Bucar, Wei Zhao, Guillaume Castagnino, Guillaume de Lafond, Herv� COMMOWICK, Hiroaki Nakamura, James Voth, Jamie Gloudon, Jarno Huuskonen, Joe Williams, Joshua M. Clulow, Julien Vehent, Justin Karneges, Kevin Hester, Kevin Musker, Kristoffer Gr�nlund, Krzysztof Piotr Oledzki, Lukas Tribus, Marc-Antoine Perennou, Mark Lamourine, Mathieu Trudel, Michael Scherer, Neil Prockter, Nenad Merdanovic, Nick Chalk, Olivier Burgard, Oskar Stolc, Patrick M�zard, Pieter Baauw, Prach Pongpanich, Rauf Kuliyev, Remi Gacogne, Sagi Bashari, Sasha Pachev, Sean Carey, Sergiy Prykhodko, Simon Horman, Simone Gotti, Stathis Voukelatos, Tait Clarridge, Thierry Fournier, Todd Lyons, Vincent Bernat, William Lallemand, William Turner, Willy Tarreau, Yuxans Yao, Yves Lafon.

Additionally, we are very thankful to a few organisations who have sponsored the development of certain advanced features which required to dedicate a person or a team for a significant amount of time (I hope I have not missed any) :

HAProxy Technologies (formerly Exceliance)
Loadbalancer.org
StackOverflow
SmartFile
SmugMug
ImageShack

Don't forget to offer a beer to your distro packagers who make your life easier. It's hard to list them all, but if you don't build from sources, you're likely running a package made and maintained by one of these people :

debian: Vincent Bernat, Apollon Oikonomopoulos, Prach Pongpanich
Fedora: Ryan O'hara
OpenSuSE: Marcus R�ckert
Other?: contact me to mention you

And last, I'd like to assign a special mention to our most active mailing list supporters during that period who make the project a reality by off- loading the support task from developers and kindly help our 800 permanent subscribers on a daily basis, BIG THANKS to you guys :

Baptiste Assmann
Lukas Tribus
Cyril Bont�
Jonathan Matthews
Thomas Heil

For the HAProxy development team here in France, it will be time to do some errands and buy some Champagne to celebrate the event :-)

June 7th, 2014 : HTTP/1.1 reloaded

Six years ago, Daniel Stenberg notified on the libcurl mailing list that Mark Nottingham, chair of the IETF HTTP Working Group, initiated a review of existing HTTP implementations. Since HAProxy was mentionned in this list, Aleks Lazic relayed this mail to the haproxy mailing list and I attempted to fill in the form with the information I could provide on HAProxy. By then HAProxy 1.3.15 had just been released without keep-alive nor HTTP/1.1 support. Being confronted to a lot of interoperability issues between clients and servers through HAProxy, I felt that it was a great opportunity to join the Working Group to try to improve the situation from my angle of view. I was impressed to see that participants work professionally and calmly. Noone really tries to defend his own implementation, what matters is interoperability, even if that sometimes needs changes in their code. Of course, the concerns are different for various types of components (eg: security, performance, complexity, etc), often resulting in long debates. And like many other products, HAProxy experienced quite a number of code changes consecutive to clarifications made to the spec. The latest example of this happened after Google Chrome users were seeing HAProxy's 408 error page on a number of web sites.

And 7 years after this positive work started (OK, 6 years for me, I started late), the group's efforts have finally resulted in 11 new RFCs! The 6 first (RFC 7230 to 7235) replace the aging RFC2616 :

RFC7230 : Message Syntax and Routing
RFC7231 : Semantics and Content
RFC7232 : Conditional Requests
RFC7233 : Range Requests
RFC7234 : Caching
RFC7235 : Authentication

The next ones are about authentication scheme and method registrations (7236 and 7237), and extensions whose discussion was initiated in the working group but which were not carried by the working group (status code 308, "Forwarded" and "Prefer" header).

If you're implementing an HTTP agent (client, server, proxy, gateway, or whatever), please read them. They cover a lot of corner cases that are encountered in field and which are the result of the fact that the protocol evolved faster in field than in the docs. The first two ones are already a very good start to demystify the protocol and get rid of your wrong beliefs.

What next ? HTTP/2.0 is on the rails and has been for about 2 years now. It's time to focus on it!

May 28th, 2014 : 1.5-dev26 : hopefully last!

This version tries to be the last one in the development cycle. There were very few changes. The agent-check was completed. A potential CPU busy loop when reloading with the systemd wrapper was fixed. A possible buffer overflow with regex in inlikely configs was fixed. The "str" match is now used by default for string samples. No major new feature is considered missing for the final release, so only fixes and minor updates may be merged now. Source code and changelog are available as usual here.

May 10th, 2014 : 1.5-dev25

Four important issues were fixed since dev24. One could cause crashes on out-of-memory. Another one concerns FreeBSD where the shared session cache could have been used without locking, causing random crashes as well. The recent fixes for HTTP request body forwarding randomly caused pauses when using balance url_param. Last, arguments "-i" and "-n" were ignored on ACLs since dev23. Some pending changes were completed as well. Half-closed timeouts are now supported. Unix sockets are supported on the server side, as well as abstract namespace sockets on Linux. This allows backends and frontend to connect together without consuming TCP ports. The old unmaintained BSD and OSX Makefiles were removed. Per-listener process binding is finally possible using the "process" keyword on "bind" lines, which makes it possible to have one stats socket per process. Version 2 of the PROXY protocol was implemented on the server side. A few other minor improvements were made. Source code and changelog are available here.

Apr 26th, 2014 : 1.5-dev24

This version fixes 3 major regressions from dev23, one causing transfers to be interrupted after the configured timeout, another one where redirects could sometimes cause a crash, and one slowing down SSL. Other minor issues were fixed. The stats page now supports chunked mode, keep-alive and compression, and may be declared in any section with no warning. Health checks can be started within a smaller delay. http-request/response now support set-map/del-map/add-acl/del-acl to add/remove pattern entries to maps and ACLs on the fly based on data extracted from the traffic. Heartbleed attacks (CVE-2014-0160) are detected and blocked even on vulnerable OpenSSL implementations. Source code and changelog are available here.

Apr 23th, 2014 : ~~1.5-dev23~~ : Broken

This new version addresses half of the remaining changes before -final : use_backend supports log-format expressions. Maps and ACLs now share the same pattern lists which are dynamically updatable from the CLI. SSL web sites now load faster thanks to dynamic record size adjustments. Compression of chunked HTTP responses was fixed and enabled again. About 35 bugs were fixed. We're getting close to 1.5-final. Source code and changelog are available here.

Feb 3rd, 2014 : 1.5-dev22

The whole polling system was finally reworked to replace the speculative I/O model designed 7 years ago with a fresh new event cache. This was needed because the OpenSSL API is not totally compatible with a polled I/O model since it stores data into its internal buffers. One of the benefits is that despite the complexity of the operation, the code is now less tricky and much safer. HTTP keep-alive is now enabled by default when no other option is mentionned, as it is what users expect by default when they don't specify anything and end up with tunnel mode. A new option "tunnel" was thus added for users who would find a benefit in using tunnel mode. After an HTTP 401 or 407 response, we automatically stick to the same server so there's normally no need for "option prefer-last-server" anymore. SSL buffer size are automatically adjusted to save round trips as suggested by Ilya Grigorik, reducing the handshake time by up to 3. The CLI reports more info in "show info" and now supports "show pools" to check memory usage. SSL supports "maxsslrate" to protect the SSL stack against connection rushes. The "tcp-check" directive now supports a "connect" action, enabling multi-port checking. Very few changes are pending before 1.5-final : ACL/map merge (being reviewed), HTTP body analyzer fixes (in progress), agent check update (started), per-listener process binding (experimentations completed). Source code is available here.

Dec 17th, 2013 : 1.5-dev21

Several early testers reported a few annoying bugs yesterday, so I preferred to fix them quickly and issue another release than wasting everyone's time on these bugs. Use this version instead of 1.5-dev20 to be safe. Please refer to the changelog for more information. Source code is available here.

Dec 16th, 2013 : 1.5-dev20 : keep-alive, finally!

This version took 6 months to be released given the difficulty of some changes, and it collected a number of new features. The most awaited of them is server-side keep-alive. The first commit for this feature was initially attempted almost 4 years ago. There are still some limitations, idle server connections are not accounted for maxconn and not reported in the stats for example. And option http-keep-alive still needs to be specified to benefit from the feature. The memory usage has significantly dropped, 640 bytes saved per idle connection on 64-bit systems. All sample fetch expressions (including ACLs) now support a list of converters applied to the sample. The new map feature allows input samples to be converted to other ones. The most common usage of this is geolocation, but it may also be used for massive redirect tables. Maps are updatable live from the CLI. Redirects support the log-format syntax and can embed some elements collected from the request. Hash algorithms can now be selected per backend. Health checks have been improved with the agent-check and tcp-check to build send/expect rules. Please refer to the changelog for more information. Source code is available here.

Jun 17th, 2013 : 1.4.24 and 1.5-dev19 : security fix

A new vulnerability was discovered in all versions from 1.4.4 and above. It was fixed today with 1.4.24 and 1.5-dev19 (CVE-2013-2175). This vulnerability can induce a crash when configs involving tail-relative header offset such as hdr_ip(xff,-1) are used. Please see the advisory for more details. All 1.4 and 1.5 users must upgrade.

Additionally, a few other important bugs were fixed. One of them was a regression introduced in 1.5-dev12, which could randomly crash haproxy on rare circumstances when using pipelined requests with a slow client. Last, some endless loops were possible since 1.4 when using consistent hashing algorithms with flapping servers.

On the positive side, many small new features were finally merged, such as http-response rule sets, ability to change task priority, DSCP field, Netfilter mark and log-level based on L7 ACLs, ability to selectively accept the PROXY protocol header using ACLs, support for environment variables in ACLs and fetches, addition of a 3rd stick-counter, filtering on the stats page, transparent proxy for FreeBSD/OpenBSD, and a few other things. Last but not least, the doc on ACLs/fetches got a major lift-up to deduplicate keywords. A few important issues still need to be addressed, and server-side keep-alive is expected as well before final 1.5 can be released, hopefull by the end of this summer.

Please refer to the 1.4 changelog and 1.5 changelog for more information. Source code is available here for 1.4 and here for 1.5.

Apr 3rd, 2013 : 1.4.23 and 1.5-dev18 : security fix

A vulnerability in all 1.4 and 1.5 releases was fixed in 1.4.23 and 1.5-dev18 (CVE-2013-1912), affecting HTTP fetches in TCP request inspection rules. All 1.4 and 1.5 users must upgrade.

In addition to this, 1.5-dev18 brings a significant number of improvements, among which use of environment variables in all addresses (bind, log, source, server, ...), agent-based health check system, support for systemd, TLS ALPN, and a few other nice features.

Please refer to the 1.4 changelog and 1.5 changelog for more information. Source code is available here for 1.4 and here for 1.5.

Apr 1st, 2013 : April Fool's

I decided to launch HAProxy in a weather ballon to get direct connectivity over WiFi to a few nearby datacenters.

Recent news...

Dec 28th, 2012 : Development 1.5-dev17

I broke a few things in dev16 which have been repaired in 1.5-dev17. This is a very small version with few changes. All 1.5-dev users are then encouraged to upgrade to dev17.

Dec 24th, 2012 : Development 1.5-dev16

Here comes 1.5-dev16. Thanks to the amazing work Sander Klein and John Rood have done at Picturae ICT we could finally spot the freeze bug after one week of restless digging ! This bug was amazingly hard to reproduce in general and would only affect POST requests under certain circumstances that I never could reproduce despite many efforts. It is likely that other users were affected too but did not notice it because end users did not complain (I'm thinking about webmail and file sharing environments for example). During this week of code review and testing, around 10 other minor to medium bugs related to the polling changes could be fixed.

Another nasty bug was fixed on SSL. It happens that OpenSSL maintains a global error stack that must constantly be flushed (surely they never heard how errno works). The result is that some SSL errors could cause another SSL session to break as a side effect of this error. This issue was reported by J. Maurice (wiz technologies) who first encountered it when playing with the tests on ssllabs.com.

Another bug present since 1.4 concerns the premature close of the response when the server responds before the end of a POST upload. This happens when the server responds with a redirect or with a 401, sometimes the client would not get the response. This has been fixed.

Krzysztof Rutecki reported some issues on client certificate checks, because the check for the presence of the certificate applies to the connection and not just to the session. So this does not match upon session resumption. Thus another ssl_c_used ACL was added to check for such sessions.

Among the other nice additions, it is now possible to log the result of any sample fetch method using "%[]". This allows to log SSL certificates for example. And similarly, passing such information to HTTP headers was implemented too, as "http-request add-header" and "http-request set-header", using the same format as the logs. This also becomes useful for combining headers !

Some people have been asking for logging the amount of uploaded data from the client to the server, so this is now available as the "%U" log-format tag. Some other log-format tags were deprecated and replaced with easier to remind ones. The old ones still work but emit a warning suggesting the replacement.

And last, the stats HTML version was improved to present detailed information using hover tips instead of title attributes, allowing multi-line details on the page. The result is nicer, more readable and more complete, as can be seen on the demo page.

All 1.5-dev users are then encouraged to upgrade to ~~dev16~~. Update: minor last-minute regression on the stats page, please use the latest snapshot instead.

Dec 12th, 2012 : Development 1.5-dev15

This is an incremental fixup on top of dev14 to address the few remaining bugs that were reported since its release, and particularly the high CPU usage that a few users have reported. Some SSL issues were also fixed and its cache was improved to use 4 times less memory. The conditions to enable compression were tightened. The strange server errors that were logged and counted for years were in fact client errors, and that was fixed. SSL handshake errors are now logged. Tracking layer 7 information is now possible ; it was limited to "src" till now. It will allow people behind proxies to benefit from some scraping or DoS protection.

All 1.5-dev users are then encouraged to upgrade to dev15.

Nov 26th, 2012 : Development 1.5-dev14

This is a quick fixup for all the bugs that were reported in dev13. All users are encouraged to upgrade to dev14 and to drop both dev12 and dev13 !

Nov 22th, 2012 : Development 1.5-dev13 with Compression!

This is the largest development version ever issued, 295 patches in 2 months! We managed to keep the Exceliance team busy all the time, which means that the code is becoming more modular with less cross-dependences, I really like this ! First, we got an amazing amount of feedback from early adopters of dev12. It seems like SSL was expected for too long a time. We really want to thank all those who contributed patches, feedback, configs, cores (yes there were) and even live gdb access, you know who you are and you deserve a big thanks for this! Git log says there were 55 bugs fixed since dev12 (a few of them might have been introduced in between). Still, this means that dev12 should be avoided as much as possible, which is why I redirected many of you to more recent snapshots. These bugs aside, I'm proud to say that the whole team did a really great job which could be summarized like this :

SSL:
- many more features ; client and server certificates supported on both sides with CA and CRL checks. Most of the information available in SSL can be used in ACLs for access control. Some information such as protocol and ciphers can be reported in the logs. These information are still not added to HTTP logs though, a lot of config work is still needed.
- cache life time and maximum concurrent SSL connections can be set. Unfortunately OpenSSL happily dereferences NULL malloc returns and causes the process to die if memory becomes scarce. So we can only limit its maximum amount of connections if we want to limit the memory it uses.
- TLS NPN was implemented with the help from Simone Bordet from Jetty, and can be used to offload SSL/TLS for SPDY and to direct to a different server depending on the protocol chosen by the client.
- Ivan Ristic from ssllabs and Andy Humphreys from Robinson-way provided very valuable help in diagnosing and fixing some nasty issues with aborts on failed handshakes and improve from an E-grade to an A-grade.
HTTP Compression
- HTTP payload compression was implemented at Exceliance to achieve bandwidth usage reduction and reduce page load time on congested or small links. Compression is extremely CPU and memory intensive, so we spent most of the time developing dynamic adaptations. It is possible to limit the maximum RAM dedicated to compression, the CPU usage threshold and bandwidth thresholds above which compression is disabled. It is even possible to adjust some of these settings from the stats socket and to monitor bandwidth savings in real time. Proceeding like this ensures a high reliability at low cost and with little added latency. I've put it on the haproxy web site with nice bandwidth savings (72% avg on compressible objects, 50% on average, considering that most downloads are compressed sources). I must say I'm very happy of this new feature which will reduce bandwidth costs in hosted infrastructures ! And it goes back to the origins of haproxy in zprox 14 years ago :-)
Health checks
- SSL is now usable with health checks. By default it is enabled if the server has the "ssl" keyword and no "port" nor "addr" setting. It can be forced using "check-ssl" otherwise. So now running an HTTPS health check simply consists in using "option httpchk" with "ssl" on the server.
- send-proxy is also usable with health checks, with the same rules as above, and the "check-send-proxy" directive to force it. The checks also respect the updated spec which suggests sending real addresses with health checks instead of sending unknown addresses. This makes it compatible with some products such as postfix 2.10 for example.
Polling
- speculative polling was generalized to all pollers, and sepoll disappeared as it was superseded by epoll. The main reason for this important change is the way OpenSSL works and the fact that it can easily get stuck with some data in buffers with no I/O event to unblock them. So we needed to engage into this difficult change. I'd have preferred to delay it to 1.6 if I was offered the choice ! But in the end this is good because it's done and it improves both performance and reliability. Even select() and poll() are now fast.
- the maxaccept setting was too low on some platforms to achieve the highest possible performance, so it was doubled to 64 and is now per listener so that it automatically adjusts to the number of processes the listener is bound to. This ensures both best performance in single process mode, and quite good fairness in multi-process mode.
Platform improvements
- Linux 3.6 TCP Fast Open is supported on listeners ("tfo" bind keyword). This is used to allow compatible clients to re-establish a TCP connection in a single packet and save one round-trip. The kernel code for this is still young, I'd be interested in any feedback.
- use of accept4() on Linux >= 2.6.28 saves one system call.
Process management
- stats socket can now be bound to specific processes. This is useful to monitor a specific process only.
- "bind-process" now supports ranges instead of silently ignoring them.
- "cpu-map" establishes a mapping between process numbers and CPU cores. This is important when running SSL offloaders on dedicated processes because you don't want them to pollute the low-latency L7 core.
Misc : "redirect scheme" makes it easier to redirect between http and https, config error reporting was improved for "bind" and "server" lines by enumerating the list of supported options dynamically.

I must say I'm much more confident in dev13 than I was with dev12 and I have already upgraded the main web site which has been upgraded every few days with recent snapshots. I've build and run it on Linux i586/x86_64/armv5/v7, OpenBSD/amd64 and Solaris/sparc without any issue anymore.

To all those running SSL tests on dev12, please drop it for dev13. I don't think we introduced regressions (but that's still possible), but I know for sure that we fixed a lot! The usual changelog and source are available at the usual place.

Sept 10th, 2012 : Development 1.5-dev12 with SSL!!!

The main, long-awaited, feature this time is native SSL support on both sides, with SNI and multi-process session sharing. The work took several months to be done at Exceliance because it required a major rewrite of the lower connection layers in order to support multiple data layers. This was a very painful task, but doing so allowed us to shrink the SSL patch from several thousands of lines of hardly maintainable code to a few hundreds of SSL-specific code. The code supports the Server Name Indication TLS extension (SNI), which consists in presenting the certificate which matches the host name requested by the client. This also works with wildcard certificates, of course. The certificates can be loaded from a directory, which makes it more convenient to load hundreds or thousands at a time. And since they are loaded into a binary tree, there is no lookup overhead even if there are hundreds of thousands, which is very convenient for massive hosting providers.

In current state, the code does not yet support checking certificates, which also means that connecting to an SSL server is only useful if the LAN is safe (in short, it's only useful if the server absolutely wants to get the connection to port 443). But the Exceliance team is actively working on this.

We took care of correctly arranging connection and data layers. Right now it's perfectly possible to chain multiple layers of haproxy servers to offload more SSL, using SSL-ID affinity and the PROXY protocol in order not to lose the client's source address. Doing this with off-the shelf hardware can result in quite a cheap SSL offloader even for huge loads. We measured 4000 TPS on SSLv3 on an Atom D510 and have not yet run the tests on larger hardware.

Among the other features in this version, we can list IPv6 transparent mode, "base" pattern/acl to match a concatenation of the Host header and the URI, "urlp_val" ACL to match a URL parameter's value, support for the "nice" keyword on "bind" lines to change the priority of sessions using this bind line (useful to limit SSL CPU impact), the ability to clear/feed stick-table entries on the stats CLI (which got lost forgotten in a dead branch), and the usual set of halog features and optims.

The changelog is available for more information, though there are a lot of commits to transform the connection layers. Users who need SSL should really give it a try. While we got a number of useful reports on the mailing list and could fix some issues, it is very likely that some bugs remain, so if you observe abnormal behaviours, please report your experiences there.

On the stable branch side, 1.4.22 was silently released one month ago with a number of small fixes and a number of minor feature improvements, such as the ability for putting a server in soft-stop mode from the stats web page in admin mode, and support for the "httponly" and "secure" flags on cookies.

June 4th, 2012 : Development 1.5-dev11

A large number of bugs were fixed again since 1.5-dev10, some of them being regressions from 1.5-dev8 and later versions. See the changelog for more information, but nobody should be running on dev9 nor dev10. Minor harmless features were added in dev11, such as new actions on the stats page, a few new cookie options, and some minor improvements on URI hashing and server recovery mode. Users should really upgrade, as I don't want to waste time trying to spot stupid bugs in configs that are notoriously broken.

May 21st, 2012 : Stable 1.4.21

A number of old bugs were reported recently. Some of them are quite problematic because they can lead to crashes while parsing configuration or when starting up, which is even worse considering that startup scripts will generally not notice it. Among the bugs that 1.4.21 fixes, we can list : risk of crash if using reqrep/rsprep and having tune.bufsize manually configured larger than what was compiled in, risk of crash when using header captures on a TCP frontend (uncaught invalid configuration), risk of crash when some servers are declared with checks in a farm which does not use an LB algorithm (eg: "option transparent" or "dispatch"), "balance source" did not correctly hash IPv6 addresses resulting in IPv4 connections to IPv6 listeners always having the same hash. Some other minor fixes and improvements were merged. While it's very likely that almost nobody is affected by the bugs above, troubleshooting them is annoying enough to justify an upgrade.

May 8th, 2012 : Development 1.5-dev9

Many new features were added since 1.5-dev7 (I forgot to announce dev8 here). Let's summarize this shortly : new logging subsystem with customizable log formats, a unique-ID generator, full rework of the buffers and HTTP message storage, merge of the ACL and pattern fetch code, ACL support for IPv6 addresses, cookies, URL parameters and arbitrary payload, support for specifying a precise occurrence in fetch functions, much better error reporting for ACL parsing errors, the long-awaited "use-server" directive, minor improvements to the error capture reports, and a significant number of bugfixes. Please give it a test.

March 10th, 2012 : Stable 1.4.20

A few bugs were reported since 1.4.19 was released, and some were found in 1.5 during development. Servers tracking disabled servers would still be used while disabled. Zero-weight servers could still dequeue requests pending in the backend's queue. The build was broken on FreeBSD since 1.4.19. Since the introduction of client keep-alive, a server would not pick a pending requests after releasing a connection if it keeps exactly maxconn-1 connections, which is problematic with low maxconn values. POST requests smaller than the buffer would experience an undesirable additional delay of 200ms due to a flag being left unconditionally enabled on the buffer. Sometimes when sending data wrapping across the buffer, haproxy would fail to merge TCP segments into a single one, which results in a few PUSH packets that can sometimes be observed during chunked-encoded transfers (this was just a missed optimization). 1.4.20 was released with all these changes. Some of them are important enough to justify an upgrade, eventhough they've been here for a very long time.

January 8th, 2012 : Stable 1.4.19

A few bugs were fixed since 1.4.18, and they impacted users so I wanted to release something now eventhough none of them is critical. First, Sagi Bashari fixed the usage of alternative header name for the "forwardfor" option. An incompatibility between server tracking and slowstart, was diagnosed by Ludovic Levesque : the weight would remain at the lowest level forever. Daniel Rankov reported that option nolinger did not work in backends. It looks like it has been the case for a very long time now. An issue in the string indexing in ebtrees was diagnosed by Julien Thomas. It is used in ACLs could theorically affect the ACL code though it has no visible effect since all patterns in the same ACL are interchangeable. Timothy Garnett reported an issue where Ruby clients were experiencing an extra delay in response time. After analyzing some network traces, it appeared that Ruby likes to send POST requests in multiple incomplete packets, waiting for the first one to be ACKed before pushing the rest, which is incompatible with the delayed ACK. Since we get the incomplete request, we can notice that it's missing data and re-enable quick ACKs to make the client send the rest ASAP. Obvously the client should be fixed as its behaviour makes it very sensible to network latency. Brian Lagoni reported that TProxy broke after Linux 2.6.34 kernel, because the address family was previously assumed to be AF_INET and was not set in HAProxy. Last bug, I was fed up with HAProxy blocking invalid server responses which were sent without headers. I finally understood that it was because some requests were sent with a "\0" in the URI which HAProxy did not block, and Apache considered the request line truncated and ignored the HTTP version, resulting in HTTP/0.9. So the request parser was modified to reject control characters in the URI (the standard forbids other characters but we can't change too much in a stable version without risking breaking some setups). One minor feature was merged. Mark Lamourine worked on a solution to send a server's name in a header when connections are established to a server. I know this can be useful in some silo-like setups and the code does not present any risk of regression so I accepted to include it in 1.4. So 1.4.19 was released with all these changes. If you have no problem with current version, there is no need to upgrade.

September 16th, 2011 : Stable 1.4.18

The fix for the space parsing in the headers that I made of 1.4.17 was not complete, because it results in negative header lengths being returned for headers that are exclusively composed of spaces. I have checked the whole code to see if this can have any nasty effect, and I couldn't find one, since everytime, we check the length before the contents (we're saved by an optimization). Still, I don't like having dangerous code lie around, especially in stable versions. I know for instance that some people apply custom patches on top of it and may get trapped. So i have issued 1.4.18 with that fix. I also included the recent patch from Finn Arne Gangstad to split domain names on ':' too, as I agree that whenever a port is specified, the host name cannot easily be checked. And I added a match for header length so that it's now a lot easier to check for an empty header. The rest are just usual doc and halog updates. I don't think there is any specific reason to rush on this new version, but if you're in the process of upgrading an older one, please avoid 1.4.17 and use 1.4.18 instead.

September 10th, 2011 : Development 1.5-dev7

Five months have elapsed since 1.5-dev6. A massive amount of changes was merged since then. Most of them were cleanups and optimizations. A number of changes were dedicated to making listeners more autonomous. The immediate effect is a more robust handling of resource saturation, and the second effect is the removal of the 10-years old maintain_proxies() function which was harming performance and hard to get over. Halog was improved too (faster with more filters). A significant number of external contributions were merged, among them the stats socket updates to clear session-table keys by values. There are too many changes to list, but nothing too dangerous, so I'd say it's the 1.5-dev version I trust the most today. Please give it a test.

September 5th, 2011 : Stable 1.4.17

Last week an issue was discovered with an application emitting spaces after the content-length value, which caused haproxy to report an error when parsing it. After some checks, it appeared that haproxy ought to ignore these spaces, so this was addressed. It was an opportunity to improve invalid request and responses captures, so that any message rejected for its malformation can be captured. A new minor feature making the X-Forwarded-For header addition conditional was added because users had to resort to complex tricks to do that. Last, halog was updated to latest version. Due to the issue with the header above, I released 1.4.17. Quite frankly most users don't need to upgrade. However it's better to use this one for new deployments.

August 6th, 2011 : Stable 1.3.26 and status updates

Previous 1.3 version was released 14 months ago, the same day as 1.4.8. Since then, a number of fixes went into 1.4 and a part of them were queued for 1.3. These fixes are not *that much* important but are still worth a release. Thus, both 1.3.26 and 1.3.15.13 were released and are available as source and precompiled binaries for Linux/x86 and Solaris/sparc.

I realized that I don't use 1.3 anymore myself, and for this reason I'm afraid of the risk of introducing regressions with future backports. So I decided that it was time to turn 1.3 into a "critical fixes only" status after 2.5 years of stable releases and 5 years of existence, meaning that minor fixes will probably never get there anymore, and that future releases, if any, will be focused on important bugs. That does not mean it's not supported anymore, but that fixes will come at a very slow pace and that new deployments are encouraged to use 1.4 if they expect a responsive support.

I'm also switching the 1.3.14 and 1.2 branches to the "unmaintained" status since nobody appears to be using them anymore (last fixes were more than 2 and 3 years ago respectively).

August 5th, 2011 : Stable 1.4.16

Since 1.4.15 was released 2 months ago, very few minor bugs were detected. They were so minor that it was worth waiting for other ones to be found, but after some time, there wasn't any point making users wait any more, so I released 1.4.16. A few minor improvements were also made based on feedback from users. Among the changes, MySQL checks now support Mysqld versions after 5.5, health checks support for multi-packet response has been fixed, the HTTP 200 status can be configured for monitor responses, a new http-no-delay option has been added to work around buggy HTTP implementations that assume packet-based transfers, chunked-encoded transfers have been optimised a bit, the stats interface now support URL-encoded forms, and halog correctly handles truncated files. There is no real emergency to upgrade.

June 7th, 2011 : Country IP Blocks needs help

Quite a few HAProxy users rely on geolocation lists freely provided by Country IP blocks, either to add a request header indicating the origin country, or to select the datacenter closest to the client. Now this nice service needs some money to continue operations otherwise they're forced to close. They're asking for donations. Their service has been offered for free with a high quality to many HAProxy users for some time now, I think it would really be fair that these users in turn help their nice provider. They need $2000 before next week, this certainly is achievable if all big site using their lists donate $100 each to keep them alive. Never forget that for any free software or service on the Net, there are always people working hard to keep the service alive and who have to pay bills at the end of the month.

May 31st, 2011 : HAProxy participates to IPv6 day

About two weeks ago I registered to participate to the World IPv6 day. The concept is very simple : on June 8th, many web sites will be available both in IPv4 and IPv6. Why only that day ? Because there exists some places where IPv6 can be resolved but not reached, causing the dual-stack sites to be unreachable from these places. By having many sites running IPv6 on the same day, network admins will notice the problem comes from their site and not from the outside since many sites will be unreachable at the same time.

HAProxy was running dual-stack a few years ago but I had to revert this due to many problem reports. Still some visitors might have noticed the little green image indicating to them if their browser can connect to IPv6. Since I noticed on the participants list that some sites were already running with dual-stack enabled, I decided to enable it here again in advance, so that I'll be able to revert it in case some visitors report any issue. If no issue is reported until June 8th, I'll probably leave that enabled.

Unfortunately, the Dedibox serving as a cache for the web site is in a network that is not yet IPv6-enabled. That's really a shame, considering that we upgraded it from an old box that was on an IPv6-capable network. I really don't understand what's happening at Free for taking that long a time to enable IPv6 on all their network segments, it does not seem to be on their top priority list. But since the site is running at home behind my Nerim internet access which has been IPv6-enabled for something like 10 years now, I could announce the ADSL endpoint address in the DNS.

Enabling IPv6 on your web site really is trivial with HAProxy. You just have to add "bind :::80" to your frontend and announce the IPv6 address as an AAAA record in your DNS zone, and that's all. No readdressing, no routing changes, nothing fancy. And you can even get IPv4/IPv6 statistics like here. BTW, I know for sure that some of the World IPv6 Day participants have done exactly that with their HAProxy config too :-)

Apr 8th, 2011 : stable 1.4.15 & 1.5-dev6

Two annoying bugs were detected on 1.4 at Exosec, one week apart. The first one limits the usable content-length to 32-bit on 32-bit platforms, despite the efforts made in the code to support 64-bit quantities everywhere. It was then fixed in 1.4.14. Yesterday, while working on the backport of 1.4 fixes to 1.3, I spotted that the patch to fix the issues with spaces in cookies that was merged in 1.4.9 introduced a regression due to a typo. In some circumstances, a malformed header sent by the server can crash haproxy when cookie-based persistence is enabled. Thus 1.4.15 was released as an emergency update to address this. The bug has never been reported because it's extremely unlikely to appear, unless a server tries to provoke it on purpose.

In the mean time, 1.5-dev4 was released with a huge amount of fixes and architectural reorganizations (too many to list here), which were needed to continue the work towards server-side keep-alive. 1.5-dev5 enabled server-side IPv6 support and fixed a number of remaining bugs. 1.5-dev6 was finally released to address the last regressions reported on the list yesterday as well as the important bug above.

Now, everyone should have understood that all users of 1.4 >= 1.4.9 or 1.5 > 1.5-dev3 must upgrade.

Please consult the 1.4 CHANGELOG and the 1.5 CHANGELOG for more information.

Mar 9th, 2011 : stable 1.4.13

Many annoying bugs were discovered both when working on 1.5-dev and by users. Some headers were not correctly processed after removal of the last header (issue reported to the list by Stefan Behte), disabling a disabled proxy from the CLI could result in a segfault (reported by Bryan Talbot, fixed by Cyril Bont�), "balance url_param" was completely broken on POST requests (reported by Bryan Talbot too), it is theorically possible to get HTTP chunk size wrong if only the CR is sent as the last byte of the buffer, waiting for the LF to wrap around in a subsequent packet, ACLs loaded from a file did not correctly close the file descriptor upon success (reported by Bertrand Jacquin), the recently added srv_id ACL could segfault if the server is not known (reported by Herv� Commowick), rlimits were not correctly updated for listening sockets (reported by the loadbalancer.org team), the stats page in admin mode did not support multi-packet requests (fixed by Cyril).

1.4.12 was released with all those fixes, and Hank A. Paulson reported a crash with pattern files with empty lines caused by a regression introduced into 1.4.11 by a fix for correctly handling empty lines. So 1.4.13 was released a few hours later to avoid any issue.

I'd like to thank all of these contributors, because well-detailed bug reports are equally important as code contributions. Once again, all users of 1.4 are encouraged to upgrade in order to avoid boring troubleshooting of stupid bugs. This time I have added Sparc builds too, as there are still requests for those. As usual, please check Changelog, with sources and Linux binaries at the usual places.

Feb 10th, 2011 : stable 1.4.11

1.4.11

critical

With all that, all users of 1.4 are strongly encouraged to upgrade. As usual, please check Changelog, with sources and Linux binaries at the usual places.

Nov 11th, 2010 : devel 1.5-dev3

Oct 29th, 2010 : stable 1.4.9

Among the issues that were fixed, a listener could be left in an unrecoverable state in case of memory shortage during an accept(). POST requests that were followed by a CRLF (forbidden) in a late packet could cause some TCP resets to be emitted on Linux due to those two unread bytes (diagnosed with Dietrich Hasselhorn). Servers that were disabled while processing requests could still drain new requests from the global queue. HTTP header handling for ACLs did not correctly consider quotes and used to consider commas within quotes as a list delimitor. A server with address 0.0.0.0 used to rely on the system to connect to this address (which is always itself). Now it forwards the connection the same way as in the transparent mode. Various error reports and logs were fixed or improved, and many doc typos were fixed.

Now concerning the improvements, Krzysztof Oledzki improved his netsnmp-perl plugin to support listening sockets, and Mathieu Trudel's Cacti templates were merged. Judd Montgomery and Cyril Bont�'s work to support setting servers up/down from the stats interface has been merged too. Gabor Lekeny added LDAPv3 health checks. Herv� Commowick improved the MySQL check to support a complete login sequence with a real username. When option "abortonclose" is set and a client disconnects while waiting for the server, now we forward the close notification to the server. That way the server can decide whether to continue or close. This is important for servers dealing with long polling requests. The Explicit Content Validation (ECV) check code was finally merged after 18 months of reviewd and fixes by various people. That was one major cause for delaying this release. Health checks can now rely on a string that is looked up in server responses. Persistence cookies now support inactivity timeouts and time to live. This is needed with some new terminals such as iPhones where the browser is never closed and the terminal sticks to the same server forever (which is particularly undesired during a partial outage). Also, we now have a new "preserve" option for cookies in "insert" mode, which indicate that if the server sets the cookie, then we let it pass unaffected. This allows servers to terminate persistence upon logout. Last, the "halog" utility was improved to support per-url and per-termination code statistics. This means that it's now trivial to know what URLs take the most processing time.

Version 1.4.9 was released with all that, with sources and Linux binaries at the usual places. Some of these fixes will slip into 1.3 too.

Oct 23th, 2010 : new httperf results : 572000 reqs/s

this interesting post

Varnish

Kristian said he used httperf. I tried it in the past but did not manage to get good numbers out of it. He said he found some "httperf secrets", so that made me want to try again. First tests were limited to approximately 50000 requests per second with httperf at 100% CPU. Something close to my memories. But reading the man, I found that httperf can work in a session-based mode with the "--wsess" parameter, where it also support HTTP pipelining. Hmmm nice, we'll be less sensible to packet round-trips :-) So I tried again with haproxy simply doing redirects. Performance was still limited to 50000 requests per second.

In fact, there appears to be a default limit of 50000 requests per second when "--rate" is not specified. I set it to 1 million and ran the test again. Result: about 158000 requests per second at 100% CPU and with haproxy at 44%. Since my machine is a Core2 Quad at 3 GHz, I fired 3 httperf against one haproxy process. The load reached a max of 572000 requests/s with an average around 450000 requests per second. This time, haproxy and all 3 httperf were using 100% CPU. What an improvement!

These tests mean nothing at all for real world uses of course, because when you have many clients, they won't send you massive amounts of pipelined requests. However it's very nice to be able to stress-test the HTTP engine for regression testing. And this will be an invaluable measurement tool to test the end-to-end keep-alive when it's finished. I still have to figure out the meaning of some options and how to make the process less verbose. Right now it fills a screen with many zeroes, making it hard to extract the useful numbers. I'm grateful to Kristian to have made me revisit httperf !

Aug 26th, 2010 : devel 1.5-dev1 with many goodies

Stack Overflow Team

donated

Also, I must say that as a software engineer, it's always a lot better to have someone explain their needs with high expectations than having to guess how a feature will be used.

Geoff Dalgas and Jeff Atwood described to me in great details what they needed to do : perform request throttling per IP address, possibly based on various criteria, in order to limit risks of service abuse. That was very interesting, because that feature was being thought about for about 4 years without enough time to completely develop it, and also the new stickiness framework that was contributed by Exceliance and Loadbalancer.org was making that really possible, although an important design rework had to be operated first within the code.

During the tests with Geoff and Kyle Brandt, it appeared that some more changes had to be operated to be able to store any criteria in the tables (eg: bandwidth per IP address), and to be able to consult and change the table contents at runtime, leading to a more and more generic code. Kyle has been very patient and comprehensive, I think I have changed the mechanisms and configuration syntax at least 5 or 6 times during the tests, but he always took the time to understand the changes and adapt his configurations. If I had been at his place, I would have got bored earlier, so I owe him a big thanks for his patience !

Now the code has been running fine in production overthere, so it's time to release it and merge it into the master branch. I won't extend further on how it works, since Kyle has put an excellent explanation on his blog that is a lot more clear than the doc (that reminds me that the architecture guide really needs some lifting).

I'll add quick status on the current code. Some core changes that I wanted to do earlier will now start. But that means that 1.5-dev1 should be as stable as 1.4.8. I'm not saying that I would suggest to anyone to push it into production, but it can clearly be used to mitigate DDoS attacks as well as stop service abuses. I could get it to stop connection floods slightly above 200000 connections per second (yes, two hundreds thousands) and let the good traffic pass through. So for this reason, I think that people who are regularly exposed to such trouble may find it useful to keep it handy.

Now what's next ? Right now the data in the tables is local to one process, so it is not shared if you start multiple processes, nor it is across reloads. The second step of the stickiness extensions developped by Exceliance and Loadbalancer.org will include stickiness table synchronization between multiple hosts. Some work will still be needed to be able to share counters, but since this development is done in a flexible way, it should not be too hard to adapt it later. BTW, I also owe a big kudos to the Git versionning system, which has made it very easy to rework my patches after every change and bugfix until they were looking good, through massive abuse of branching and rebasing.

Too much talk. The code is available here, the ChangeLog is here and the doc is here. Since this is a development version, no binary is provided.

The last words naturally go to the really cool guys at Stack Overflow. It's very nice to see some sites and companies involve time and money and take risks to make Open Source products better. Of course they benefit from this work, but at no point during the whole development did they try to reduce the focus to their specific needs, quite the opposite. From the very first exchanges, their goal clearly was to make the product better, and that must be outlined. That's now achieved and I really appreciate their involvement. Thank you guys !

June 16th, 2010 : stable 1.4.8 and 1.3.25

stick-table

monitor-net

Version 1.4.8 was released, with sources, Linux and Solaris binaries at the usual places. Version 1.3.25 was released too with a few other pending fixes, and sources and Linux/Solaris binaries at the usual places too.

Users of 1.3 after 1.3.16, or 1.4 with TCP only frontends will likely want to upgrade, eventhough if they have not been hit yet, they will probably never be. Users of stick-tables will have to upgrade too. The fact that we're now spotting bugs that have been there for two years is an encouraging sign of stability.

June 10th, 2010 : stable 1.4.7

failed req

denied req

dispatch

http_proxy

Some people will like the new halog, as it is able to report per-server stats on status codes, error ratios, average connection times and response times, which is very handy for quick checking of your prod's health.

The complete ChangeLog for version 1.4.7 is here and the sources are here.

On a side note, someone asked on the list how haproxy would perform on a Guruplug Server Plus. I ordered one a few months ago and finally received it. Well, it's a disaster, not only it is slower than the Geode, but it heats so much that metal parts hurt to the touch and power supplies fry after a few months. Definitely not something to buy. Check the full review on previous link if you're interested.

May 16th, 2010 : stable 1.4.6

srv_is_up()

May 13th, 2010 : stable 1.4.5

version 1.4.5

here

First, Cyril Bont� provided the new ignore-persist directive. it allows haproxy to ignore the persistence cookie on some requests which validate an ACL-based condition. It is particularly suited to optimise the load balancing of static or stateless objects in the middle of a stateful farm.

Second, it was planned 3 years ago to be able to feed ACLs with large data sets loaded from files, but it was still not implemented due to the lack of precise needs. Now, 3 years later, more and more people are reporting difficulties writing large configurations, and the last config I saw which was 104000 lines long convinced me that it was urgent to support this feature. But matching requests against very large datasets can be CPU intensive, so I have extended my Elastic Binary Trees to support new lookup methods and now it is possible to lookup a string or an IP address among tens of thousands in a few tens of nanoseconds. This means that it is now possible to use haproxy to perform geolocation. For instance, checking that a source address belongs to one of the 38400 european networks only consumes 2% CPU at 40000 requests per second.

The rest are just minor improvements. Tt's now possible to stick on an IP address extracted from an HTTP header, and I improved a bit more the halog analyser, which is now possible to report request counts by status codes. It also gained some nice performance boost as it can now parse about 1.3 Gigabytes of logs per second on a 3 GHz Core2.

I expect that this version will take some time to spread because it only contains new features and will likely not be backported to various distros. Still, some power users will probably interested in giving it a try.

April 7th, 2010 : stable 1.4.4

option http-pretend-keepalive

option http-server-close

Jetty's HTTP implementation seems to be the flakiest though. It even manages to send "HTTP/1.1 100 Continue" intermediate responses when the client sends "Expect: 100-continue", but it closes the connection just after that message, resulting in a 502 error for the client.

Cyril also fixed an issue with appsession where a cookie whose name begins like the appsession cookie could be mistaken for it. Those issues were enough to justify a new release.

Very few other minor fixes were brought there, and a minor feature was added. It consists in being able to bind to a source address found in a header when connecting to a server. Normally this will be the X-Forwarded-For header. This requires use of the Linux kernel TPROXY patches, and makes it possible for backend servers to see the initial client's IP even when several layers of proxies have been passed through.

March 30th, 2010 : stable 1.4.3 and 1.3.24

were fixed

backported

March 17th, 2010 : stable 1.4.2

these issues

error-limit

on-error

release 1.4.2

March 5th, 2010 : stable 1.4.1

Other issues

release 1.4.1

Also, Solaris users will now be happy, I unpacked and replugged my Ultra5 so the Sparc binary is available again.

On a side note, I have removed the link to the haproxy.org mirror because it has been outdated for the last 6 months and even remained 1 week on an expired DNS zone. I failed several times to contact Kevin Kuang there, so I don't even know who manages it now if any. If someone gets in touch with him, please ask him to contact me.

February 26th, 2010 : New stable branch 1.4 !

version 1.4 is now released

the ChangeLog

January 28th, 2010 : stable 1.3.23

1.3.22

request-learn

force-persist

http-check send-state

1.3.23

minor features

January 25th, 2010 : 1.4-dev7 & 1.4-dev8

Exceliance

Loadbalancer.org

Update:

January 16th, 2010 : 4-hour network outage

here

Nerim

SFR

January 8th, 2010 : 1.4-dev6

will

default-server

1.4-dev6

January 2nd, 2010 : New year, new features

committed

HTTP keep-alive and pipelining

"option httpclose"

defaults

frontend

backend

"option http-server-close"

maxconn

1.4-dev5

snapshot

Formilux

decent load time savings

this page

Stay tuned...

October 18th, 2009

matrix of all known bugs

1.3.15.2

1.3.16

1.3.17

doomed

1.3.15.X before 1.3.15.7

1.3.19

1.3.21

risk

1.3.14.14

1.3.15.10

1.3.20

good

1.3.22

no known bug

October 14th, 2009 : 1.3.22

causing a crash when connecting to the stats socket

1.3.22

Please don't use 1.3.21 and update to 1.3.22!

October 12th, 2009 : 1.4-dev4, 1.3.21

reset counters

change a server's weight

stats page

exact status

consistent hashing

static round-robin

1.3.21

September 26th, 2009

a new page

in the latest snapshot

September 24th, 2009 : 1.4-dev3 + sponsors

a new clean snapshot

Compared to latest stable 1.3.20 version, 1.4-dev3 provides new features, among which support for the CLF log format, RDP protocol load-balancing and persistence, a new interactive CLI, an improved HTML stats page, support for inspecting HTTP contents in TCP frontends and switching to HTTP backends (allowing HTTP+SSL to coexist on the same port), support for forcing of the TCP MSS on frontends, smart network optimizations to reduce the number of TCP packets in a session, runtime-configurable buffer sizes, support for more than 64k concurrent connections, config parser support for "no option xxx" to disable options that were enabled by default, and correct 1xx status code processing. Developments to support keep-alive have already started, and if time permits, SSL integration will be attempted. The code looks amazingly stable for the amount of changes, and will probably not change much anymore, so any bug found in this version must be reported and fixed. Also, new feature submissions should be based on this version. It will be easier to implement for submitters and for me to merge.

Several large sites are already running on 1.4-dev2 with great success. This one should be even better, but given the number of changes, it should be monitored more closely at the beginning.

Last, I have a very good news that I hope will give ideas to others : Exceliance and Loadbalancer.org have both agreed to contribute some manpower and money to implement the complete persistence framework that everyone is dreaming about into haproxy. That's a tough work and I'm not certain it will be ready for 1.4 (though it might, depending if I'm as late on my releases as usual). I would personally like to thank them both for their contributions. When you have to put your money in commercial solutions, please never forget to consult first the guys who involve time and money in opensource projects, because they are the ones who help the projects evolve and live !

August 9th, 2009 : 1.3.20

transfer.ro

Another bug was reported by Romuald du Song, who found that option tcplog would log using global parameters if no logger was defined. It can be either helpful or annoying. This is now fixed and a warning is emitted when such a configuration is encountered, so that people running off erroneous configs can easily fix them.

This time I expect 1.3.20 to be the good one. It's always a good sign when we fix minor bugs or recent regressions introduced by bug fixes. 1.4-dev2 has also been released to help people track changes in the two versions in parallel.

July 27th, 2009 : 1.3.19

slowloris

new kinds of bugs

Exceliance

1.3.19

Since some of the bugs above were present in earlier versions, a new release was emitted for 1.3.15 and 1.3.14 too for the late users who have not upgraded yet. I really think it's the last one for 1.3.14. 1.3.15 might still see another one till the end of the year, and that will probably be all for this one after 20 months of free support :-)

The first development version should be released too, but I need to update my release scripts first, they are inadequate and take me too much time to use, so stay tuned !

June 27th, 2009 : HAProxy to counter DoS attacks

announcement of the Slowloris

Indeed, HAProxy does not need a new thread nor process to accept a new connection, it only needs some RAM (16-32 kB per connection). Some people are already using it past 70000 concurrent connections, which cannot be achieved on Apache which needs an expensive thread or process per connection. More specifically, HAProxy will only forward complete and valid requests. This means that it will not bother Apache while the attacker is playing with its few thousands connections, and all valid requests will immediately pass through. And the icing on the cake is that HAProxy can kill requests which take too much time to complete, using timeout http-request (more than a few seconds is not to be considered normal).

Once again, we observe a derivate use of a load-balancer, which is a bit expected : when a tool is designed to accept 10 times more load than the servers it feeds, there is nothing surprizing that it can be used to protect them ! Let's see if Apache evolves towards providing more tunables to mitigate such attacks... In the mean time, a drop-in anti-DoS configuration is available here.

May 10th, 2009 : 1.3.18

Rocket Fuel Inc

struct session

struct requri

During a troubleshooting session with the T20 guys (Maxim Fedchishin, Jason Coward and Viktor Brilon from modX team, Hans from RightScale team), I came across an old leftover process doing nothing after a soft-reload. That issue is brought once in a while by various people, but it happens too rarely for anyone to get an opportunity to debug it. The guys accepted that I installed a debugger on their machine to see what the process was doing. It was deadlocked in free() during the reload. And that made sense : during a reload, the old process releases as much memory as possible to leave room for the new one. If the two signals sent by the second one are too close to each other, the second signal is sent while the first one has not completed releasing memory and we can have a recursion in the libc's free(), causing a deadlock. That has been fixed by implementing asynchronous signal delivery. Thank you guys for giving me the opportunity to catch that rare event!

Problems aside, a few minor features were merged. The stats are now more readable, report max session rates and provide full 64-bit counters everywhere. It is now possible to forward invalid requests or responses without blocking them, but they will still be captured. The config parser now warns about possibly unwanted ordering of ACLs or reqxxx/rspxxx. Several wrong printf() format strings have been fixed. The build process now supports an alternative architecture, and the RPM spec file has been cleaned. A new balance hdr(header) algorithm has been added to balance depending on a header hash. A new option enables addition of the destination IP address in the X-Original-To header. And last but not least, the doc has been massively cleaned up and reorganised.

With all these fixes, I released 1.3.18, as well as 1.3.15.9 and 1.3.14.13 which are probably among the last ones of their respective branches after 12 and 18 months of maintenance.

April 19th, 2009 : new performance record broken !

new series of benchmarks again

10Gig

20% CPU

100000

March 29th, 2009 : 1.3.17

who's.amung.us

Thanks Bart!

other minor fixes and enhancements

1.3.17

March 22th, 2009 : 1.3.16 !

1.3.16

March 10th, 2009 : 1.3.16 is getting closer !

1.3.16

March 9th, 2009

1.3.15.8 and 1.3.14.12

"track"

December 4th, 2008

1.3.15.7 and 1.3.14.11

October 12th, 2008

That was enough to release 1.3.15.5 and 1.3.14.9. I recommend that any user of 1.3.14 or 1.3.15 upgrades, as these fixes present very minor risk and fix really annoying problems.

September 14th, 2008

the mailing list

showing the problem

1.3.15.4 and 1.3.14.8

maxconn=1

September 3rd, 2008

connection regulation mechanism

maxconn

has been posted on 37signals

September 2nd, 2008

1.3.15.3 and 1.3.14.7

"balance url_param ... check_post"

July 20th, 2008 : two lines...

TCP content inspection

stop half of the spams

		tcp-request inspect-delay  35s
		tcp-request content reject if REQ_CONTENT

June 21th, 2008

haproxy versions 1.3.15.2 and 1.3.14.6

Since it was a design problem, it took a lot of time analyzing the root cause and finding a solution. However, as a positive side effect, the fix now makes the redispatch option work for requests which overflow a queue. That way, clients do not get a 503 error anymore but can be served by another server (which was the purpose of the redispatch option.

Note that it is possible that 1.2 is also affected by the issue since some parts of the faulty code have not changed since. But it is very hard to determine if it is faulty or not, and backporting the fix would take even more time. Maybe I will eventually take a look at it if people complain about the issue.

Update (2008/06/28): Alexander Staubo, who first noticed the problem, has run a new series of tests showing that the problem is definitely fixed. It also demonstrates the very nice positive effect of running with maxconn 1 with Rails servers.

May 25th, 2008

haproxy versions 1.3.15.1 and 1.3.14.5

sepoll

accept()

April 19th, 2008

haproxy version 1.3.15

stats updates

enhancements of server checks

tracking

dynamic intervals

leastconn

transparent

connection failures

redirects

documentation updates

ChangeLog

Once again, a lot of code comes from contributions. I'd like to specially thank Krzysztof Oledzki for a lot of useful contributions, including the SNMP agent, and the guys at Nokia for the good work they have done on POST parameter hashing.

March 30th, 2008

I finally assembled my new machines and installed the donated 10-Gig Myricom NICs. I ran a few benchmarks. Result: new bandwidth records set for HAPoxy: 9.897 Gbps and 35128 hits/s! It's possibly the highest bitrate achieved to date with an opensource load-balancer! BTW, even most commercial ones are commonly limited to 4 Gbps by hardware design. What's a bit frustrating for a precision-tweaker like me is that those NICs work out-of-the box on dirt cheap hardware, there's almost no joy passing beyond the first 4 Gbps :-)

March 8th, 2008

haproxy maintenance version 1.3.14.3

February 23rd, 2008

I finally decided to buy an expensive motherboard to upgrade my PC in order to begin testing with the 10Gig NICs. I selected an Asus P5E3-WS Pro because I needed PCI-X slots for my older cards. I've put a Core2 Duo E8200 (45nm) because I wanted a lot of silence. The mobo has 2 PCI-E 16x slots, which made it possible for me to run a back-to-back test between two 10Gig NICs. Since the board does not support PCI graphics cards, I had to boot on serial port (the only VGA card I've got running on this mobo was a cheap crappy GeForce 8400 GS which does not work under X). Well, my first test is quite encouraging : I can achieve 10 Gbps of HTTP traffic between the two NICs with the server and client on the same machine, which means that the hardware will be able to support haproxy under the same conditions. I tried with client + haproxy + server but the bit rate diminished to about 6-7 Gbps. I'm impatient to buy the 3 other mobos to build a full lab. I will mix 2 Athlons and one C2D so that I can experiment which one is better for which type of traffic. Stay tuned!

January 21th, 2008

haproxy maintenance version 1.3.14.2

sepoll

truncated responses

GNU Makefile

rewritten

full variable compatibility

Distribution packagers are encouraged to migrate over to this one

new configuration manual

keywords and all their options

minor robustness and performance tuning parameters

January 13th, 2008

full transparent proxy

without touching either source nor destination addresses

December 12th, 2007 : Santa Claus left a present for me at EXOSEC !

Some of you might have already got their hands on this. For those who don't know yet, this beautiful piece of art is a 10 Gbps Ethernet NIC from Myricom. For a long time, I had been tempted by their legendary high performance network cards, which were said everywhere to be able to saturate a 10 Gig wire under Linux without putting too much stress on the CPU, using a mainstream opensource driver, and without resorting to dirty tricks such as TOE. What would a performance addict like me need more ?

I finally decided to mail these guys and described how I'm currently used to benchmark HAproxy with aggregated Gigabit NICs, with a minimum of 4 NICs in a setup (1 for the client, 2 for the proxy, 1 for the server). 4 hours later, when I woke up, I had a mail from Charles Seitz, Myricom's CEO. He explained to me that he was pleased to offer me 4 NICs with cables, plus one spare of each just in case, as their contribution to the project... yes, I'm talking about a donation of five 10Gig NICs! That's awsome! And if it would not be enough for some of you to find them really cool, he also provided me with french-speaking contacts, free access to their support and important advices for the choice of motherboards to get the best out of those wonderful NICs! I don't even know the polite words to say in such circumstances :-)

Today I've been monitoring the shipping steps at UPS. This evening, I noticed that they arrived at EXOSEC. After leaving the customer's, I went back there to find this big parcel on my desk, with its contents very carefully packed. I must say that I was both very excited and extremely careful while opening the packaging.

The first thing I noticed after extracting the first NIC from its packaging was that it had a very clean design, as can be seen on this photo. They are also very thin as shown on the picture on the right, so there will be no problem putting two of them side-by-side in the proxy.

The CX4 connector looks a bit fragile, but careful manipulation is the minimal requirement to use the highest speed standard Ethernet. From what I understood, this is the same connector as used on Infiniband, except that 10GE has terminators on the board.

Well, obviously, there are very nice companies out there who deserve to be talked about! Their very generous support to open source projects leaves many others far behind. People say that Santa Claus lives in the North Pole, but now I know he lives in Arcadia in California :-)

Thank you very much Charles, thanks very much Myricom. Be sure to read about my first test results here.

December 6th, 2007

haproxy version 1.3.14

mailing list

dynamic server weights

slow start

graceful shutdown

servers state to outer components

multi-site architectures

BGP

stable state

October 18th, 2007

haproxy maintenance version 1.3.12.3

new version 1.3.13

mailing list

Doug Lea's dlmalloc

September 22th, 2007

September 20th, 2007

haproxy version 1.3.12.2

timeouts and retry counts

sanity checks

erroneous configurations to be accepted

license has been clarified

stable

All version 1.3 users are encouraged to upgrade to 1.3.12.2

September 5th, 2007

haproxy version 1.3.12.1

crashes under Linux

speculative I/O

nosepoll

global

stats refresh <interval>

stats page automatically refresh

new configuration manual

July 15th, 2007

Configuration Manual

all configuration keywords

examples of ACLs

content switching

examples/

June 17th, 2007

haproxy version 1.3.12

integration of ACL

Content Switching

customize your error responses

support 1.3.12

demand for a stable version

some of them even pay

June 3rd, 2007

haproxy version 1.3.11.4

2 long-standing bugs

May 14th, 2007

haproxy version 1.3.11.3

bug

random timeouts

New in this release are a better timer management and a new memory manager which is able to self-manage its pools and free unused ones when memory is becoming scarce. It is also easier to code with this new one since it's not necessary anymore to declare pool sizes. Overall, yet another performance boost of 5% has been gained.

May 10th, 2007

haproxy version 1.3.10.2

May 9th, 2007

haproxy version 1.3.10.1

serious bug

random timeouts

upgrade

-ds

nosepoll

global

May 9th, 2007

haproxy version 1.3.10

ACL

SMTP health checks

URI hashing

rbtree

performance boost around 5%

The speculative I/O processing in 1.3.9 has introduced some bugs which have been fixed in this version. I feel confident that latest changes have brought their pile of bugs too. I will probably spend some time soon to do cleanup and stabilization work, eventhough both are not really compatible.

I also want to thank all the people who contribute code and testing. You are more and more at each release. I'm impatient to clean up the remains of the old code, so that even more people can contribute code. Interestingly, all contributions till now were of high quality. This is probably induced by some sort of selection caused by the technical aspect of the product, and the skills required to use the development version. Thanks again to you all !

Apr 22th, 2007

EXOSEC

haproxy 1.3.9

graph

Apr 15th, 2007

haproxy version 1.3.9

modularization

support for FreeBSD kqueue()

A new concept was introduced too : speculative I/O. It is a new method consisting in reducing the number of calls to the expensive epoll_ctl() and epoll_wait() by attempting to access the file descriptors before being notified about their readiness. This provides an overall speed boost of 10%, which is quite much for just a poller.

Apr 3rd, 2007

haproxy version 1.3.8.2

1.3.6

the process to crash

All users of 1.3.6 and later must upgrade

Apr 1st, 2007

haproxy version 1.3.8.1

option httpclose

VIM script

Mar 25th, 2007

haproxy version 1.3.8

Just like with every release, several code optimization have led to small but noticeable performance increases, particularly on very high data bandwidth. The configuration errors are handled more gracefully now with indications about what failed and hints to resolve the issue. HAProxy now builds on MacOS 10.4 thanks to Dan Zinngrabe who provided a makefile. Also, it is now possible to send health checks to an alternate server address, thanks to a patch from Fabrice Dulaunoy.

Users of 1.3 are encouraged to upgrade to 1.3.8 as it both fixes known bugs and converges towards something less tricky than previous versions.

Mar 17th, 2007

haproxy version 1.2.17

rbtree scheduler

user

group

uid

gid

X-Forwarded-For

The architecture manual was updated to reflect new features in branch 1.2, with examples for stunnel and for load mitigation.

Users of 1.2.16 with high loads are encouraged to upgrade to 1.2.17 as it offers them the high performance of branch 1.3 with the reliability of the stable branch 1.2.

Jan 27th, 2007

haproxy version 1.3.7

critical bug

All 1.3.6 users MUST upgrade to 1.3.7

Jan 22th, 2007

haproxy version 1.3.6

HTTP message parser

carefully hand-optimized 28-states FSM

goto

Firefox

Freshmeat

microsecond

500000 times a second!

The request code has been cleaned up a lot and adapted to this new FSM. Adding layer7 rules based on new criteria is now trivial. The response code will be ported next, but the code was so much cleaner and faster that it was worth releasing one version before breaking everything. Several bugs were fixed since 1.3.5. I really consider 1.3.6 as the most likely reliable 1.3 release to date.

Jan 7th, 2007

Linux Layer7 Switching project

kernel TCP splicing

The second great news is that Sin Yu has provided me with a useful patch for the second time : the task scheduler is now based on an rbtree and not on the dirty old dual-linked list anymore. It means that people who had performance problems and who had to set all their timeouts to the same value as a workaround will not have to do this anymore. I have tested, and the code works like a charm ! Thanks again Sin !

Jan 2nd, 2007

haproxy version 1.3.4

Content-Switching features !!!

It is now possible to select a backend (server pool + load balancing algo) depending on any parameter in the request, such as any part of the URI, the host name, etc.... As of now, I've merged Sin Yu's patch to permit switching based on a request regex, but the framework is ready to accept many other criteria. The HTTP request parser has been completely rewritten to support unlimited header inspection, and the statistics page has been rewritten, as can be seen on the demo page. It is far from being finished right now, but it seems pretty usable. The server state machine should be adapted though.

There is still no doc, so please note that old configurations do still work, and that in order to switch from an instance to another backend, you need to use "reqisetbe <regex> <new_proxy>". Also, there's a config example here that will be worth any doc.

Dec 5th, 2006

load balancing article

LinuxFR

tc

Jul 4th, 2006

development branch 1.3

LGPL

available here

⇐ Back to HAProxy

Contacts

Feel free to contact me at for any questions or comments :

Main site : http://1wt.eu/
e-mail :

HAProxy

The Reliable, High Performance TCP/HTTP Load Balancer

Quick links

News

Contacts